The present invention relates to a method and apparatus for testing software and hardware and, more particularly, to a distributed, expandable automated testing system which allows users operating computers to test software and hardware on test machines which are in communication with the users"" computers via a network.
The software testing phase is a critical phase in the software development process. During the software development process, the software testing phase occurs after the software has been designed, implemented in a programming language, and tested to a limited degree. During the testing phase, software testers test the software extensively to ensure that the software meets all of the requirements it is intended to meet. In order to accommodate simultaneous testing of several different software packages by several testers, multiple test machines are often implemented. Different types of software packages may need to be tested on different types of test machines, such as, for example, test machines with different hardware configurations and/or different operating systems. When a large number of testers are required to share common resources for software testing, provisions must be made for scheduling the tests in order to efficiently manage these shared resources. The efficient management of these shared resources may also require that tests and the results of the tests be recorded so that the tests can be used repeatedly if needed and so that the results of the tests can be analyzed and subsequently used for comparison with the results of tests performed at a later time.
In an effort to maximize efficiency in the handling of test scheduling and test execution, attempts have been made to automate software testing by using a server to manage test machines and to allocate test packages among the test machines in accordance with a schedule. Sun Microsystems, Inc. has proposed an automated task-based scheduler for use with UNIX platform systems which allows users operating xe2x80x9cclientxe2x80x9d machines to schedule tests to be executed on xe2x80x9ctargetxe2x80x9d machines. A central server receives a request from a client machine to perform a task. The server maintains information relating to all currently scheduled tasks on all target machines in a xe2x80x9cstatusxe2x80x9d database. The server maintains information relating to the expected duration of each test package and other test package attributes in a xe2x80x9cpackagesxe2x80x9d database. When the server receives a request to perform a task from a client machine, the server determines the loads on each of the target machines based on the expected duration of each test package and then schedules the task on the target machine with the least current load. A task file created at the client machine and copied to the server includes priority information relating to the task requested by the client machine. The target machine selects a task to be preformed based on this priority information. Once a task is completed, the results are copied back to the server which compares them to a set of xe2x80x9cgolden resultsxe2x80x9d and creates a comparison report which is mailed back to the user that requested the test.
One disadvantage of the system proposed by Sun Microsystems is that the system cannot be easily expanded. If a test machine is added to the system, the xe2x80x9cmachinesxe2x80x9d database will need to be manually updated to include information relating to the new test machine, such as its operating system and identity. Also, once a test machine is added, no provision is made for automatically rescheduling tasks taking the new test machine into consideration. This reduces the flexibility and adaptability of the system. Furthermore, if a task fails after performance of the task has begun, or if a test machine fails after tasks have been scheduled on it, all of the affected tasks will need to be rescheduled.
Another disadvantage of the system proposed by Sun Microsystems is that it is not a distributed system. The system proposed by Sun Microsystems is a local area network (LAN) having a client/server arrangement which uses scripts to communicate instructions between the users"" client machines and the central server and between the central server and the target machines. No provision is made for allowing users located remotely with respect to the testing system to access the system because no communications protocol is provided for encapsulating and transporting the scripts. Also, the test system proposed by Sun Microsystems also has no provision for automatically configuring a test machine and automatically installing an operating system on a test machine. Rather, the test system proposed by Sun Microsystems only allows a user to select a test machine already having the operating system required for executing a particular test. Therefore, if a test machine having the necessary operating system for testing the software is not available, the test cannot be executed.
Accordingly, a need exists for an automated testing system which is automatically expandable to allow new test machines to be easily and automatically added to the system, and which is distributed such that the users and components of the test system may be distributed over any network, such as the Internet.
In accordance with the present invention, an automated testing system is provided which is capable of being distributed over a network, such as the Internet, for testing hardware and software. A plurality of users operating computers interface to the automated testing system via user interfaces, which preferably are graphical user interfaces. Each user interface displays test parameter choices to the user from which the user may select test parameters relating to a software or hardware test to be performed. Each user interface generates data packets in response to selections from the user and outputs the data packets onto the network. Generally, the data packets output from the user interface comprise information relating to test parameters selected by the user, commands indicating that performance of a test is being requested, and an address of the location to which the packet is being sent. A dispatcher machine located on the network and designated by the address contained in the data packet receives the data packets and updates a list of tests to be performed.
A plurality of test machines are in communication with the dispatcher machine via the network. When a test machine is available, the available test machine generates availability data packets which indicate that the test machine is available to perform a test. Each of the availability data packets contains an address of a dispatcher machine. These availability data packets are sent over the network and routed to the dispatcher machine designated by the address contained in the availability data packet. Upon receiving an availability data packet, the dispatcher machine determines whether one or more of the tests on the list of tests maintained by the dispatcher machine are capable of being performed by the test machine which generated the availability data packet. If one or more of the tests listed are capable of being performed by the available test machine, the dispatcher machine instructs the test machine to perform one of the tests. This aspect of the present invention renders the automated testing system extremely adaptable in that it allows the system to be easily expanded or contracted because the system does not pre-allocate tests, but rather allocates tests in response to requests for work sent from the test machines to the dispatcher machines.
In accordance with the preferred embodiment of the present invention, once the dispatcher machine receives an availability data packet from a test machine, the dispatcher machine determines whether any of the tests contained on the list of tests maintained by the dispatcher machine is capable of being performed on the available test machine and, if so, prioritizes these tests and instructs the available test machine to perform the test with the highest priority. The test machine then performs the test. Preferably, the test machine obtains the test and any archives associated with the test to be performed from a library which is in communication with the user interfaces and with the test machines via the network. However, the test can be obtained from any location which is in communication with the network.
When requesting performance of a test, a user can query the library via the user interface for information relating to tests and select test parameters relating to a test to be performed based on the information provided by the library to the user interface. Information relating to the user""s selection will be contained in the data packets output onto the network to the dispatcher machine.
Each test machine has a launcher program installed thereon which provides the application program interface between the test machines and the dispatcher machine and between the library and the test machines. The launcher program is responsible for preparing the test machine to execute a test, obtaining the test and any archives associated with the test from the library or from some other location on the network, installing the test and any: associated archives on the test machine, preparing the test and archives to be executed, causing the test machine to execute the test, and outputting the results of the test to a predetermined location. In accordance with the preferred embodiment of the present invention, the results of the tests are output to the dispatcher machine. The users can query the dispatcher machines via the user interfaces to obtain status information about the tests.
In accordance with the preferred embodiment of the present invention, the automated testing system of the present invention comprises an installer machine which is capable of automatically configuring the boot ROM settings on a test machine and/or installing operating systems on a test machine. To accomplish this, a multiplexer/demultiplexer is connected to an RS232 port or ethernet local area network (LAN) port of each of the test machines and to the network. The installer machine is in communication with the dispatcher machines via the network. The installer machine receives a data packet from a dispatcher machine indicating that a particular test machine is to be reconfigured and/or have an operating system installed on it. In response to receiving the data packet, the installer machine causes the test machine to be reconfigured and/or installed via either the RS232 or-LAN port of the particular test machine.
Once the test machine has been reconfigured and/or installed, the launcher program and any other necessary software is installed on the test machine. When the launcher program is installed, the launcher program will notify all of the dispatcher machines with which it is allowed to communicate that the test machine is on the system. The notified dispatcher machine will then update a machines file contained in the dispatcher machine with information relating to the test machine, such as the address of the test machine, its hardware configuration, which users are allowed to use it, and whether it can have its boot ROM reconfigured and/or have a new operating system installed on it.